mHC 如何使用其残差流：选择性路由与近恒等混合

文章背景与核心概要

本文深入探讨了训练好的机器学习模型如何利用流形约束超连接（manifold-constrained Hyper-Connections, mHC），该技术将单一的残差路径扩展为了 \(n\) 个流。通过分析 DeepSeek-V4-Flash 的四流残差路径，作者研究了读/写路由的集中度、跨流残差混合以及流间余弦相似度。研究结果表明，模型仅使用了 mHC 所提供的理论容量的一部分：单个模块通常仅依赖大约两个流，后期的残差混合带来的可衡量收益微乎其微，且特定位置的结构对指标的影响远大于逐 token 的变化。

论文元数据 (Paper Metadata)

arXiv 标识符: arXiv:2609.05309 [cs.LG]
学科领域: 机器学习 (cs.LG), 人工智能 (cs.AI)
提交日期: 2026年9月4日
作者: Pengxiang Zhao, Xing Li, Xianzhi Yu, Wei Guo, Zhenhua Dong
许可协议: 知识共享署名 4.0

摘要 (Abstract)

超连接及其流形约束变体 mHC 将残差路径从一条流扩展到 \(n\) 条，然而训练好的模型如何利用这种容量仍不清楚：模块读取和写入的范围有多广、残差路径混合流的强度如何，以及这些流是否携带不同的表征。我们在 DeepSeek-V4-Flash 的四流残差路径中，通过有效流计数、跨流残差权重和流间余弦相似度检查了这些属性。读/写路由是集中的，但在不同深度有所变化：一个典型的注意力或 FFN 站点有效地使用了大约两个流，而主导流在各层之间发生变化，且表征在方向上保持着明显的差异。残差混合较为适度，主要发生在早期层；在第 22-42 层中，该路径主要将每个流分别向前推进。有针对性的干预实验确立了这些模式的功能意义。用恒等映射替换后期混合器仅使 C4 困惑度增加 1.9%，并保持了六个任务的平均得分，而替换早期混合器则使困惑度增加 41%。将每个早期混合器固定为其 C4 诊断均值，仅使困惑度增加 0.2%，并将平均得分降低 0.25 个百分点，这表明在其评估的指标上，特定位置的结构比逐 token 的变化更重要。同样，在每个站点保留每个 token 的三个最大路由权重，最多使困惑度增加 2.7%，并将平均得分改变最多 0.4 分。因此，所研究的模型仅实现了四流 mHC 所提供的部分灵活性：单个模块很少需要全部四个流，并且后期残差混合提供的衡量收益微乎其微。

Hyper-Connections and their manifold-constrained variant mHC widen a residual pathway from one stream to n, yet how trained models use this capacity remains unclear: how broadly blocks read and write, how strongly the residual pathway mixes streams, and whether the streams carry distinct representations. We examine these properties in the four-stream residual pathway of DeepSeek-V4-Flash using effective stream counts, cross-stream residual weights, and inter-stream cosine similarity. Read/write routing is concentrated but varies across depth: a typical attention or FFN site effectively uses about two streams, while the dominant stream changes across layers and the representations remain directionally distinct. Residual mixing is modest and occurs primarily in early layers; in layers 22-42, the pathway mostly carries each stream forward separately. Targeted interventions establish the functional significance of these patterns. Replacing the late mixers by identity increases C4 perplexity by only 1.9% and preserves the six-task average score, whereas replacing the early mixers increases perplexity by 41%. Fixing each early mixer to its C4 diagnostic mean increases perplexity by only 0.2% and reduces the average score by 0.25 percentage points, showing that its site-specific structure matters more than its token-wise variation on the evaluated metrics. Likewise, retaining the three largest routing weights per token at every site increases perplexity by at most 2.7% and changes the average score by at most 0.4 points. Thus, the studied model realizes only part of the flexibility afforded by four-stream mHC: individual blocks rarely require all four streams, and late residual mixing provides little measured benefit.

核心发现与见解 (Key Findings & Insights)

集中的路由 (Concentrated Routing): 注意力机制和 FFN 模块有效地使用了大约两个流，而不是全部四个流的容量。

Concentrated Routing: Attention and FFN blocks effectively use about two streams rather than the full capacity of four.
适度的早期混合 (Modest Early Mixing): 残差混合主要发生在网络的早期层，而第 22–42 层几乎完全将每个流独立地向前推进。

Modest Early Mixing: Residual mixing happens primarily in the earlier layers of the network, whereas layers 22–42 carry each stream forward almost completely separately.
后期混合器的影响极小 (Minimal Impact of Late Mixers): 用恒等映射替换后期混合器仅导致 C4 困惑度微增 1.9%，且下游任务性能没有损失。

Minimal Impact of Late Mixers: Replacing late mixers with identity mappings yields only a minor 1.9% increase in C4 perplexity with no loss to downstream task performance.
特定位置的结构 vs. Token 变化 (Site-Specific Structure vs. Token Variation): 将早期混合器固定为其诊断均值表明，特定位置的架构配置远比动态的逐 token 变化更为关键。

Site-Specific Structure vs. Token Variation: Fixing early mixers to their diagnostic means demonstrates that site-specific architectural configuration is far more crucial than dynamic token-wise variation.